Display control apparatus, display control method, and recording medium

ABSTRACT

A display control apparatus includes: a head mounted display information acquisition block configured to acquire information .indicative of a position of a head mounted display worn on a head of a user; an input apparatus information acquisition block configured to acquire information indicative of a position of an input apparatus for use by the user; an instruction input acquisition block configured to acquire information indicative of an instruction entered in the input apparatus; and an image generation block configured to generate an image to be displayed on the head mounted display on a basis of an instruction acquired by the instruction input acquisition block. The image generation block displays an image of the input apparatus at a position in the image determined on a basis of a relative position with the head mounted display of the input apparatus.

BACKGROUND

The present disclosure relates to a display control technology and, moreparticularly, to a display control apparatus, and a display controlmethod that are configured to control display on a head mounted display,and a non-transitory computer-readable recording medium.

Game playing is practiced in which, with a head mounted display that isconnected to a game machine worn on the head of a user, the useroperates a controller for example while looking at a screen displayed onthe head mounted display. With an ordinary stationary display connectedto a game machine, the user's visual field range extends also to theoutside of a display screen, so that the user may not concentrate on thescreen or lose the sense of absorption in the game being played. On thecontrary, wearing a head mounted display makes the user look at only thevideo shown on the display, thereby providing effects of the increasedsense or absorption and enhanced game entertainment.

SUMMARY

The inventors hereof recognized that, in order to allow more and moreusers to enjoy games in which head mounted displays are used, displaycontrol technologies of higher convenience are required.

In carrying out the disclosure and according to one mode thereof, thereis provided a display control apparatus. The display control apparatusincludes: a head mounted display information acquisition blockconfigured to acquire information indicative of a position of a headmounted display worn on a head of a user; an input apparatus informationacquisition block configured to acquire information indicative of aposition of an input apparatus for use by the user; an instruction inputacquisition block configured to acquire information indicative of aninstruction entered in the input apparatus; and an image generationblock configured to generate an image to be displayed on the headmounted display on a basis of an instruction acquired by the instructioninput acquisition block. The image generation block displays an image ofthe input apparatus at a position in the image determined on a basis ofa relative position with the head mounted display of the inputapparatus.

According to another mode thereof, there is provided a display controlmethod for a computer, including: acquiring information Indicative of aposition of a head mounted display worn on a head of a user; acquiringinformation indicative of a position of an input apparatus for use bythe user; acquiring information indicative of an instruction entered inthe input apparatus; generating an image to be displayed on the headmounted display on the basis of an acquired instruction; and displayingan image of the input apparatus at a position in the image determined ona basis of a relative position with the head mounted display of theinput apparatus.

According to a further mode thereof, there is provided a non-transitorycomputer-readable recording medium recording a display control programfor a computer, the program including: by a head mounted displayinformation acquisition block, acquiring information indicative of aposition of a head mounted display worn on a head of a user; by an inputapparatus information acquisition block, acquiring informationindicative of a position of an input apparatus for use by the user; byan instruction input acquisition block, acquiring information indicativeof an instruction entered in the input apparatus; and by an imagegeneration block, generating an image to be displayed on the headmounted display on a basis of an instruction acquired by the instructioninput acquisition block. The image generation block displays an image ofthe input apparatus at a position in the image determined on a basis ofa relative position with the head mounted display of the inputapparatus.

It should be noted that any combinations of the components describedabove and those obtained by translating the expressions of the presentinvention between method, apparatus, and system are also valid asembodiments of the present invention.

As described and according to the embodiments of the present disclosure,user convenience of a head mounted display is enhanced.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective views illustrating a usage environment of a gamesystem practiced as one embodiment of the present disclosure;

FIG. 2 is an external perspective view illustrating a head mounteddisplay practiced as another embodiment of the present disclosure;

FIG. 3 is a functional configuration diagram illustrating the headmounted display;

FIGS. 4A and 4B are diagrams illustrating external views of an inputapparatus;

FIG. 5 is a diagram illustrating a configuration of a game machine;

FIG. 6 is a functional configuration diagram illustrating the gamemachine;

FIG. 7 is a diagram illustrating one example of a first game image thatis displayed on the head mounted display;

FIG. 8 is a diagram Illustrating another example of the first game imagethat is displayed on the head mounted display;

FIG. 9 is a diagram illustrating still another example of a first gameimage that is displayed on the head mounted display;

FIG. 10 is a diagram illustrating yet another example of a first gameimage that is displayed on the head mounted display;

FIG. 11 is a diagram illustrating a different example of a first gameimage that is displayed on the head mounted display;

FIG. 12 is a diagram illustrating a still different example of a firstgame image that is displayed on the head mounted display;

FIG. 13 is a diagram illustrating a yet different example of a firstgame image that is displayed on the head mounted display;

FIG. 14 is a diagram illustrating a separate example of a first gameimage that is displayed on the head mounted display;

FIG. 15 is a diagram illustrating a still separate example of a firstgame image that is displayed on the head mounted display;

FIG. 16 is a diagram illustrating one example of a second game imagethat is displayed on a display apparatus;

FIG. 17 is a flowchart indicative of a procedure of a display controlmethod practiced as still another embodiment of the present disclosure;

FIG. 18 is a flowchart indicative of a procedure of the display controlmethod practiced as still another embodiment of the present disclosure;and

FIG. 19 is a flowchart indicative of a procedure of the display controlmethod practiced as still another embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following describes a display technology based on a head mounteddisplay (HMD). A head mounted display is a display apparatus that aremounted on the head of a user such that the eyes of the user arecovered, thereby allowing the user to view still images and movingimages displayed on a display screen arranged in front of the user'seyes. The subjects to be displayed on the head mounted display may besuch content as movies or television programs; in the present;embodiment, an example is described in which a head mounted display isused as a display apparatus that displays game images.

Now, referring to FIG. 1, there is shown an environment in which a gamesystem 1 practiced as one embodiment of the present disclosure is used.The game system 1 includes a game apparatus 10 for executing a gameprogram, an input apparatus 6 through which user instructions areentered in the game apparatus 10, an imaging apparatus 14 for takingpictures of a real space around a user, a head mounted display 100 fordisplaying a first game image generated by the game apparatus 10, and adisplay apparatus 12 for displaying a second game image generated by thegame apparatus 10.

The game apparatus 10 executes a game program on the basis ofinstruction input entered from the input apparatus 6 or the head mounteddisplay 100 and the position or attitude of the input apparatus 6 or thehead mounted display 100, and generates a first game image to transmitthe generated first game image to the head mounted display 100 andgenerates a second game image to transmit the generated second image tothe display apparatus 12.

The head mounted display 100 displays the first game image generated inthe game apparatus 10. Further, the head mounted display 100 transmitsinformation related with a user input operation done through an inputapparatus arranged on the head mounted display 100 to the game apparatus10. The head mounted display 100 may be connected to the game apparatus10 with a wired cable or in a wireless manner such as wireless localarea network (LAN).

The display apparatus 12 displays the second game image generated in thegame apparatus 10. The display apparatus 12 may be a television having adisplay and a speaker or a computer display.

The input apparatus 6 has a function of transmitting a user instructioninput to the game apparatus 10; in the present embodiment, the inputapparatus 6 is configured as a wireless controller that provideswireless communication with the game apparatus 10. The input apparatus 6and the game apparatus 10 may establish wireless connection by use of aBluetooth (trademark) protocol. It should be noted that the inputapparatus 6 may also be a wired controller connected to the gameapparatus 10 with a cable, in addition to a wireless controller.

The input apparatus 6, driven by a battery, is configured by having twoor more buttons for performing instruction input operations for making agame progress. When the user operates a button of the input apparatus 6,an instruction input done by this operation is transmitted to the gameapparatus 10 through wireless communication.

The imaging apparatus 14 is a video camera configured by a chargecoupled device (CCD) imaging device or a complementary metal-oxidesemiconductor (CMOS) imaging device for example and takes pictures of areal space with a predetermined period, thereby generating a frame imagefor each period. The imaging apparatus 14 is connected to the gameapparatus 10 through an interface such as universal serial bus (USB) orthe like. In image captured by the imaging apparatus 1A is used toderive the positions and attitudes of the input apparatus 6 and the headmounted display 100 in the game apparatus 10. The imaging apparatus 14may be a ranging camera for obtaining distance or a stereo camera. Inthis case, the imaging apparatus 14 allows the acquisition of distancesbetween the imaging apparatus 13 and the input apparatus 6 or the headmounted display 100 for example.

In the game system 1 of the present embodiment, the input apparatus 6and the head mounted display 100 each have a light emission block thatemits a light of two or more colors. During a game, the light emissionblock emits light in color indicated by the game apparatus 10 which isimaged by the imaging apparatus 14. The imaging apparatus 14 takes apicture of the input apparatus 6 and generates a frame image, supplyingthe generated frame image to the game apparatus 10. Acquiring the frameimage, the game apparatus 10 derives the positional information aboutthe light emission block in the real space from the position and size ofthe image of the light emission block in the frame image. The gameapparatus 10 handles the positional information as a game operationinstruction, thereby reflecting the game operation instruction onto suchprocessing of the game as controlling the operation of a player'scharacter.

In addition, the input apparatus 6 and the head mounted display 100 eachhas an acceleration sensor and a gyro sensor. The sensor detectionvalues are transmitted to the game apparatus 10 with a predeterminedperiod. Receiving the sensor detection values, the game apparatus 10acquires the attitude information of the input apparatus 6 and the headmounted display 100 in the real space. The game apparatus 10 handles theattitude information as an operation instruction for a game and reflectsthe attitude information onto the processing of the game.

Referring to FIG. 2, there is shown an external view of the head mounteddisplay 100 practiced as one embodiment of the present disclosure. Thehead mounted display 100 has a main body section 110, a head contactsection 112, and a light emission section 114.

Arranged on the main body section 110 are a display, a GPS unit foracquiring positional information, an attitude sensor, and acommunication apparatus. The head contact section 112 may include abiometric information acquisition sensor for detecting biometricinformation such as user's body temperature, heartbeat, bloodcomposition, perspiration, brainwave, and brain blood flow. The lightemission selection 113 emits light in a color indicated by the gameapparatus 10 as described above and functions as the standard forcomputing the position of the head mounted display 100 in an image takenby the imaging apparatus 14.

The head mounted display 100 may further include a camera for takingpictures of user's eyes. The camera arranged on the head mounted display100 allows detection of field of view, pupil movement, and blink of theeyes of the user.

In the present embodiment, the head mounted display 100 is described;however, the display control technology of the present embodiment isalso applicable not only to the head mounted display 100 in a limitedsense but also a head mounted display having eye glasses, aspectacle-type display, a spectacle-type camera, a headphone, a headset(a headphone with a microphone), an earphone, an earring, an ear-hookedcamera, a hat, a hat with camera, or a hair band, for example.

Referring to FIG. 3, there is shown a functional diagram of the headmounted display 100. The head mounted display 100 has an input interface122, an output interface 130, a backlight 132, a communication controlblock 140, a network adaptor 142, an antenna 144, a storage block 150, aGPS unit 161, a wireless unit 162, an attitude sensor 164, an externalinput/output terminal interface 170, art external memory 172, a clockblock 180, a display apparatus 190, and a control block 160. Thesefunctional blocks are also realized by only hardware, only software, ora combination thereof.

The control block 160 is a main processor that processes signals such asan image signal and a sensor signal, instructions, and data and outputsthe results of the processing. The input interface 122 receives anoperation signal and a setting signal from an input button or the likeand supplies the received signals to the control block 160. The outputinterface 130 receives an image signal from the control block 160 andmakes the display apparatus 190 display the received image signal. Thebacklight 132 supplies backlight to a liquid crystal display that makesup the display apparatus 190.

The communication control block 140 transmits data entered from thecontrol block 160 to the outside in a wired or wireless manner throughthe network adaptor 142 or the antenna 144. Also, the communicationcontrol block 140 receives data from the outside in a wired or wirelessmanner through the network adaptor 142 or the antenna 144 and outputsthe received data to the control block 160.

The storage block 150 temporarily stores data, parameters, and operationsignals that are processed by the control block 160.

The GPS unit 161 receives positional information from a GPS satellite byfollowing an operation signal from the control block 160 and suppliesthe received positional information to the control block 160. Thewireless unit 162 receives positional information from a wireless basestation by following an operation signal from the control block 160 andsupplies the received positional information to the control block 160.

The attitude sensor 164 detects attitude information such as thedirection and tilt of the main body section 110 of the head mounteddisplay 100. The attitude sensor 164 can be realized by appropriatelycombining a gyro sensor, an acceleration sensor, and an angularacceleration sensor.

The external input/output terminal interface 170 is an interface for theconnection of peripheral devices such as a USB controller. The externalmemory 172 is an external memory such as a flash memory.

The clock block 180 sets time information on the basis of a settingsignal received from the control block 160 and supplies time data to thecontrol block 160. Referring to FIG. 4, there is shown an external viewof an input apparatus. FIG. 4A shows an external configuration of thetop view of the input apparatus. The user holds a left-side hold section78 b by the left hand and a right-side hold section 78 a by the righthand and operates the input apparatus 6. Arranged on top of the housingof the input apparatus 6 are a direction key 71, analog sticks 77 a and77 b and four-types of operation buttons 76 that make up the inputblock. Four-types of buttons 72 through 75 are identified by differentcolors and different symbols. Namely, the circle button 72 has a redcircle, the cross button 73 has a blue cross, the square button 74 has apurple square, and a triangle button 75 has a green triangle. On top ofthe housing, a touch pad 79 is arranged on a flat area between thedirection key 71 and the operation button 76. The touch pad 79 sinkswhen the user presses the panel and returns to the original positionwhen the user releases the panel, thus functioning also as a pressbutton.

A function button 60 arranged between the two analog sticks 77 a and 77b. The function button 80 is used to power on the input apparatus 6 and,at the same time, activate the communication function forinterconnecting the input apparatus 6 and the game apparatus 10, Afterconnection of the input apparatus 6 with the game apparatus 10, thefunction button 80 is also used to display a menu screen on the gameapparatus 10.

A SHARE button 81 is arranged between the touch pad 73 and the directionkey 71. The SHARE button 81 is used to enter a user instruction for anoperating system (OS) or the system software of the game apparatus 10.An OPTIONS button 82 is arranged between the touch pad 79 and theoperation button 76. The OPTIONS button 82 is used to enter a userInstruction for an application (a game) that is executed on the gameapparatus 10. The SHARE button 81 and the OPTIONS button 82 may beformed as push buttons.

FIG. 4B shows an external configuration of a side of the rear section ofthe input apparatus. On top of the side of the rear section of thehousing of the input apparatus 6, the touch pad 79 extends from the topof the housing; below the side of the rear section of the housing, alight-emitting block 85 that is long sideways is arranged. Thelight-emitting block 85 has red (R), green (G), and blue (B) lightemitting diodes (LEDs) that are turned on according to emitted lightcolor information transmitted from the game apparatus 10.

On the side of the rear section of the housing, an upper button 83 a andan upper button 83 b are arranged symmetrically along longitudinaldirection, and a lower-button 84 a and a lower button 84 b are arrangedsymmetrically along longitudinal direction. The upper button 83 a andthe lower button 84 a are operated by the index finger and the middlefinger of the right hand of the user, respectively; the upper button 83b and the lower button 84 b are operated by the index finger and themiddle finger of the left hand of the user, respectively. As shown,arranging the light-emitting block 85 between the line of the upperbutton 83 a and the lower button 84 a of the right side and the line ofthe upper button 83 b and the lower button 84 b of the left side makesthe light-emitting block 85 visible without being hidden by the indexfinger or the middle finger that operates these buttons, therebyallowing the imaging apparatus 14 to suitably image the turned-onlight-emitting block 85. The upper buttons 83 a and 83 b may beconfigured as a push button and the lower buttons 84 a and 84 b may beconfigured as a pivotally supported trigger button.

Referring to FIG. 5, there is shown a configuration of the gameapparatus 10. The game apparatus 10 has a main power supply button 20, apower supply ON LED 21, a standby LED 22, a system controller 24, aclock 26, a device controller 30, a media drive 32, a USB module 34, aflash memory 36, a wireless communication module 38, a wiredcommunication module 40, a sub system 50, and a main system 60.

The main system 60 has a main central processing unit (CPU), a memoryand a memory controller that form a main storage unit, and a graphicsprocessing unit (GPU) and so forth. The GPU is mainly used forcomputational processing of game programs. These functions areconfigured as system-on-chips which may be formed on a single chip. Themain CPU has a function of executing game programs recorded to anauxiliary storage apparatus

The sub system 50 has a sub CPU and a memory and a memory controllerthat form a main storage apparatus and so forth, but does not haveeither a GPU or a function of executing game programs. The number ofcircuit gates of the sub CPU is lower than the number of circuit gatesof the main CPU and the operation power dissipation of the sub CPU islower than that of the main CPU. The sub CPU also operates while themain CPU is in a standby state and the processing function of the subCPU is restricted to lower the power dissipation.

The main power supply button 20 that is arranged on the front side ofthe housing of the game apparatus 10 and is an input block through whichuser operation input is executed, is operated to power on/off the mainsystem 60 of the game apparatus 10. The power supply ON LED 21 is turnedon when the main power supply button 20 is turned on and the standby LED22 is turned on when the main power supply button 20 is turned off.

The system controller 24 detects the pressing of the main power supplybutton 20 by the user. When the main power supply button 20 is pressedwith the main power supply being off, the system controller 24 acquiresthis pressing operation as “on instruction,” while, when the main powersupply button 20 is pressed with the main power supply being on, thesystem controller 24 acquires this pressing operation as “offinstruction.”

The clock 26 is a realtime clock that generates current date and timeinformation and supplies the generated current date and time informationto the system controller 24, the sub system 50, and the main system 60.

The device controller 30 is configured as a large-scale integratedcircuit (LSI) that executes transfer of information between devices likea south bridge. As shown, the device controller 30 is connected todevices such as the system controller 24, the media drive 32, the USBmodule 34, the flash memory 36, the wireless communication module 38,the wired communication module 40, the sub system 50, and the mainsystem 60. The device controller 30 absorbs the difference in electricproperties and data transfer speeds between the devices and controls thetiming of data transfer.

The media drive 32 is a drive apparatus which is loaded with a read onlymemory (ROM) medium 44 recording application software such as games andlicense information to drive the ROM medium 44, thereby reading programsand data from the ROM medium 44. The ROM medium 44 is a read-onlyrecording medium such as an optical disc, a magneto-optical disc, or aBlu-ray disc, for example.

The USB module 34 is a module for providing connection with externaldevices through a USB cable. The USB module 34 may provide connection tothe auxiliary storage apparatus 2 and the imaging apparatus 14 through aUSB cable. The flash memory 36 is an auxiliary storage apparatus thatconfigures an internal storage. The wireless communication module 38provides wireless communication with the input apparatus 6 for exampleon the basis of a communication protocol such as Bluetooth (trademark)protocol or IEEE802.11 protocol. It should be noted that the wirelesscommunication module 38 may be compliant with the third-generationdigital mobile phone scheme based on the international mobiletelecommunication 2000 (IMT-2000) specified by internationaltelecommunication union (ITU) or, further, may be compliant with digitalmobile phone schemes of other generations. The wired communicationmodule 40 provides wired communication with external devices, anexternal network via APS for example.

Referring to FIG. 6, there is shown a functional configuration of thegame apparatus 10. The main system 60 of the game apparatus 10 has acontrol portion 310 and a data hold portion 360. The control portion 310has a game control block 311, an instruction input acquisition block312, a taken image analysis block 313, an HMD information acquisitionblock 314, an input apparatus information acquisition block 315, a firstimage generation block 316, arid a second image generation block 317.

The data hold portion 360 holds program data of games that are executedon the game apparatus 10 and various kinds of data that are used by gameprograms.

The instruction input acquisition block 312 acquires, from the inputapparatus 6 or the head mounted display 100, information associated withuser instruction input accepted by the input apparatus 6 or the headmounted display 100.

The taken image analysis block 313 acquires an image taken by theimaging apparatus 14 and analyzes the acquired image, thereby computingthe positions and attitudes of the input apparatus 6 and the headmounted display 100. The taken image analysis block 313 acquires frameimages from the imaging apparatus 14 at a predetermined imaging speed(30 frames/second for example) and extracts the images of the lightemission blocks of the input apparatus 6 and the head mounted display100 from the frame images, thereby identifying the positions and sizesof the light emission blocks in the frame images. For example, the takenimage analysis block 313 identifies the gravity center coordinates andradius of the image of a emitting body in the frame images. When theuser turns on the light emission block in a color seldom used in anenvironment where the user uses the game system 1, the precision imagesof the light emission blocks can be extracted from the frame images.

The taken image analysis block 313 derives the positional information ofthe input apparatus 6 and the head mounted display 100 as viewed fromthe imaging apparatus 14 from the position and size of the image of anidentified light emitting body. The taken image analysis block 313derives the positional coordinates in a camera coordinates from thegravity center coordinates of the image of the light emitting body andthe information about distance from the imaging apparatus 14 from theradius of the image of the light emitting body. These positionalcoordinates and distance information form the positional information ofthe input apparatus 6 and the head mounted display 100.

The HMD information acquisition block 314 acquires informationassociated with the attitude of the head mounted display from the headmounted display 100. In addition, the HMD information acquisition block314 acquires the information associated with the position of the headmounted display 100 from the taken image analysis block 313. Thesepieces of information are transmitted to the game control block 311. Theinformation associated with the attitude of the head mounted display 100may be obtained by the analysis by the taken image analysis block 313 ofthe taken image of the head mounted display 100.

The input apparatus information acquisition block 315 acquires theinformation associated with the attitude of the input apparatus 6 fromthe input apparatus 6. In addition, the input apparatus informationacquisition block 315 acquires the information associated with theposition of the input apparatus 6 from the taken image analysis block313. These pieces of information are transmitted to the game controlblock 311. The information associated with the attitude of the inputapparatus 6 may be obtained by the analysis by the taken image analysisblock 313 of the taken image of the input apparatus 6.

If the input apparatus 6 is not imaged by the imaging apparatus 14because the input apparatus 6 gets out of the imaging range of theimaging apparatus 14 or hidden by the user body or some obstacle, thenthe input apparatus information acquisition block 315 computes theposition of the input apparatus 6 on the basis of the information about;the position of the input apparatus 6 obtained last and the attitude ofthe input apparatus 6 obtained subsequent to that moment. For example,on the basis of the data of translational acceleration obtained from anacceleration sensor of the input apparatus 6, a shift from the positionof the input apparatus 6 obtained last may be computed, therebycomputing the current position of the input apparatus 6. While the inputapparatus 6 is not imaged by the imaging apparatus 14, the positions ofthe input apparatus 6 are sequentially computed in the same manner. Ifthe input apparatus 6 is imaged by the imaging apparatus 14 again, thepositions of the input apparatus 6 sequentially computed by theacceleration data may not indicate the correct position due to theaccumulated drift errors, so that the position of the input apparatus 6newly computed by the taken image analysis block 313 may be used as thecurrent position of the input apparatus 6. This holds the same with theposition of the head mounted display 100.

The game control block 311 executes game programs and makes a gameprogress on the basis of the user instruction input acquired by theinstruction input acquisition block 312 and the information associatedwith the positions or attitudes of the input apparatus 6 or the headmounted display 100. In a game field configured by a virtualthree-dimensional space, the game control block 311 changes thepositions of player's characters subject to operation on the basis ofinputs through the direction key 71 and the analog sticks 77 a and 77 bof the input apparatus 6 and changes in the position of the inputapparatus 6 or the head mounted display 100.

The first image generation block 316 generates an image to be displayedon the head mounted display 100. The first, image generation block 316sets a viewpoint position on the basis of the position of a subject ofoperation that is controlled by the game control block 311, sets asightline direction on the basis of the attitude of the head mounteddisplay 100, and renders a virtual three-dimensional space, therebygenerating the image of a game field. The first image generation block316 relates the attitude of the head mounted display 100 with thesightline direction in the game field with a predetermined timing and,subsequently, changes the sightline direction as the attitude of thehead mounted display 100 changes. Consequently, the user can look outover the game field by actually moving his or her head, therebyexperiencing as if the user actually existed in the game field. Thefirst image generation block 316 adds game-related information and animage or the like to be displayed on the head mounted display 100 to thegenerated game field image, thereby generating a first image. The firstimage generated by the first image generation block 316 is transmittedto the head mounted display 100 through the wireless communicationmodule 38 or the wired communication module 40.

The second image generation block 317 generates an image to be displayedon the display apparatus 12. If the same image as an image to bedisplayed on the head mounted display 100 is displayed on the displayapparatus 12, the first image generated by the first image generationblock 316 is also transmitted to the display apparatus 12. If an imagedifferent from an image to be displayed on the head mounted display 100is displayed on the display apparatus 12, if a user wearing the headmounted display 100 and a user viewing the display apparatus 12 executea match game, for example, then the second image generation block 317sets a viewpoint position and a sightline direction that are differentfrom those with the first image generation block 316 and executesrendering of a virtual three-dimensional space, thereby generating animage of a game field. The second image generation block 317 addsgame-related information and an image to be displayed on the displayapparatus 12 to the generated game field image so as to generate asecond image. The second image generated by the second image generationblock 317 is transmitted to the display apparatus 12 through thewireless communication module 38 and the wired communication module 40.

On the basis of a relative position between the input apparatus 6 andthe head mounted display 100, the first image generation block 316 candisplay an image of the input apparatus 6 for the first image. It isdifficult for the user wearing the head mounted display 100 to viewsurrounding situations in the real world; however, displaying an imageof the input apparatus 6 at a position in the first image where theinput apparatus 6 would be visible unless the user wears the headmounted display 100 allows the user to have & feeling as if he or shewere viewing a real world, thereby enhancing the reality of the firstimage and providing a deeper immersive sense to the user.

When the game control block 311 changes the positions of a subject ofoperation in a game field, the first image generation block 316 canchange viewpoint position setting methods in accordance with theposition of a game field. For example, the first image generation block316 may set. a viewpoint position at a predetermined position in therear of a subject of operation in an area of a game field and makes theviewpoint position follow as the subject of operation moves; on anotherarea, the first image generation block 316 may fix a viewpoint positionat a predetermined position regardless of the change in the position ofa subject of operation. Consequently, a game field image can begenerated by appropriately setting a viewpoint position in accordancewith game field positions, thereby providing a user interface of easyoperation so as to enhance user convenience.

The first image generation block 316 can generate a first game image fora first user who views the head mounted display 100 and the second imagegeneration block 317 can generate a second game image for a second userwho views the display apparatus 12. The first user is unable to view thesecond game image displayed on the display apparatus 12 and the seconduser is unable to view the first game image displayed on the headmounted display .100, so that use of this game system allows theexecution of a match game while hiding own information from the playmate.

The following describes the above-mentioned features by way of examplesof various games.

Shooting Game:

Referring to FIG. 7, there is shown an example of the first game imagethat is displayed on the head mounted display 100. FIG. 7 shows an imageof a game in which targets are shot down with throwing knives thrownfrom the input apparatus displayed a first game image. In a game fieldwhere two or more targets 502 are arranged, the game control block 311moves the arranged targets 502 in a predetermined manner. The firstimage generation block 316 sets a viewpoint position data at apredetermined position in the game field and, on the basis of theattitude of the head mounted display 100, sets a sightline direction,thereby rendering the game field so as to generate the first game image.At this moment, on the basis of a relative position between the inputapparatus 6 and the head mounted display 100, the first image generationblock 316 arranges the input apparatus 6 in the game field and displaysan image 500 of the input apparatus 6 in the first image. The image 500of the input apparatus 6 is generated on the basis of the real inputapparatus 6 and displayed with a precision of a degree at which theinput apparatus 6 is recognizable by the user to be a real one. When achange occurs in the position and attitude of the input apparatus 6, thefirst image generation block 316 also changes the position and attitudeof the image 500 of the input apparatus 6 in the first game image.Consequently, the reality of the image of the input apparatus 6 that isdisplayed in the first game image can be enhanced.

Referring to FIG. 8, there is shown an example of the first game imagethat is displayed on the head mounted display 100. When the user shakeshis or her head horizontally or vertically to change the attitude of thehead mounted display 100, the first image generation block 316 changesthe sightline direction in accordance with the attitude of the headmounted display 100. When a target 502 enters a predetermined range 504at the center of the first game image, the game control block 311 lockson this target 502 as a target and displays on the target 502 a figure506 indicative that the target 502 is locked on. Namely, when the userviews the target 502 by directing his or her head in the direction oftarget 502 displayed in the first game image, the target 502 in thedirection of a predetermined range from that direction is locked on.Thus, setting a character or an object in the direction in which theuser is viewing as a subject of operation allows the determination orchange of subjects of operation with ease without use of the inputapparatus 6 for example, thereby enhancing the user convenience.

Referring to FIG. 9, there is shown an example of the first game imagethat is displayed on the head mounted display 100. When the userexecutes a flick operation on the touch pad 79 of the input apparatus 6,the first image generation block 316 displays a manner in which athrowing knife 508 is thrown from the position of. the touch pad 79 inthe image 500 of the input apparatus 6 displayed in the first game imagein accordance with a moving distance of the flick operation. At thismoment, a user's hand may be displayed in the first image or, so as toenhance the visibility of the throwing knife, the user's hand may be notdisplayed. Thus, changing the image 500 of the input apparatus 6displayed in the first game image in accordance with instruction inputentered in the input apparatus 6 allows the feedback of the instructionentered by the user in the first game image so as to indicate thereflection of the instruction in an easy-to-understand manner. Further,this setup gives a feeling to the user that a throwing knife comes outof the input apparatus 6 actually held by the user, so that the user canexperience a unique feeling in which a real world and a virtual worldmelt together.

Referring to FIG. 10, there is shown an example of the first game imagethat is displayed on the head mounted display 100. When the movingdistance of a flick operation done by the user on the touch pad 79 ofthe input apparatus 6 exceeds a predetermined value, the game controlblock 311 throws a throwing knife 508 in a direction determined on thebasis of the attitude of the input apparatus 6 from a virtual positionof the input apparatus 6 at a speed corresponding to the speed of theflick operation, in a game field. The first image generation block 316generates a first game image indicative of a manner in which thethrowing knife 508 is thrown. The game control block 311 computes alocus 509 of the throwing knife 508 on the basis of physical computationand moves the throwing knife 508 in accordance with the computed locus509. If the throwing knife 508 hats one of the targets 502, the gamecontrol block 311 drops the hit target 502. As described above, if atarget is locked on, the game control block 311 does not move thethrowing knife along the computed locus, but recalculates the locus soas to the throwing knife travels toward the target, thereby moving thethrowing knife along the recalculated locus. In this case, the throwingknife hits the locked-on target, thereby dropping this target.

Racing Game:

Referring to FIG. 11, there is shown an example of the first game imagethat is displayed on the head mounted display 100. This diagram shows animage of a racing game in which players ride bicycles and runs along apredetermined course, thereby competing for the time required to reach agoal. The game control block 311 makes the bicycles run in accordancewith the input entered through buttons and the like on the inputapparatus 6 in a game field where a predetermined course is arranged andcontrols the travel direction of the bicycles in accordance with theattitude of the input apparatus 6. In addition, the game control block311 executes control such that, when the position of the head mounteddisplay 100 gets lower, the air resistance to the body of the userpedaling the bicycle gets smaller, so that the speed of the bicycle isincreased; conversely, when the position of the head mounted display 100gets higher, the air resistance gets greater, so that the speed of thebicycle is decreased. In this game too, the image 500 of the inputapparatus 6 is displayed in the first game image and the front image ofthe bicycle is displayed such that the image 500 of the input apparatus6 comes to the center of a handle 510. As the position and attitude ofthe input apparatus 6 change, the first image generation block 316changes the position and attitude of the image 500 of the inputapparatus 6 in the first game image and then changes the position andattitude of the front image of the bicycle.

In this racing game, the input apparatus 6 is regarded as the handle ofthe bicycle; however, if the position of the input apparatus 6 largelydeviates from the initial position, the position of the handle gets toan unnatural position, thereby canceling the premise that the handle beoperated as the input apparatus 6. Therefore, if the position of theinput apparatus 6 deviates from a predetermined range with the positionof the input apparatus 6 at a predetermined timing used as standardbefore a race begins or during a race, the game control block 311notifies the user of that fact by displaying thereof in the first gameimage, outputting sound, or vibrating the input apparatus 6 for example,thereby prompting the user to return the position of the input apparatus6 to the correct position. At this moment, the game control block 311may stop accepting the instruction input associated with bicycle travelor stop the bicycle. By deleting the front image of the bicycle from thefirst game image or displaying a manner in which the input apparatus 6is shifted from the center of the handle of the bicycle, the first imagegeneration block 316 may notify the user that the input apparatus 6 isno more functioning as the handle of the bicycle. If the position of theinput apparatus 6 deviates from the reference position by more than afirst range, then the game control block 311 may notify the userthereof; if the position of the input apparatus 6 deviates from thereference position by more than the second range greater than the firstrange, then the game control block 311 may display a manner in which theinput apparatus 6 gets out of the bicycle without accepting theinstruction input from the input apparatus 6. Consequently, the inputapparatus 6 can foe operated within a proper range, thereby preventingthe deviation from the imaging range of the imaging apparatus 14 or theinability of displaying the image of the input apparatus 6 into thefirst game image.

Referring to FIG. 12, there is shown an example of the first game imagethat is displayed on the head mounted display 100. The first imagegeneration block 316 displays current speed 512 of the bicycle at theposition of the touch pad 79 of the image 500 of the input apparatus 6displayed in the first game image. This setup allows displaying as if adisplay apparatus were arranged although there is no display apparatusarranged on the touch pad 79 of the real input apparatus 6. In addition,this setup allows displaying of information to be presented to the userin the game, in a novel form.

Action Game:

Referring to FIG. 13, there is shown an example of the first game imagethat is displayed on the head mounted display 100. This diagram shows animage of. a game in which a player's character 516 is operated and movedinside a game field, thereby leading the player's character 516 to apredetermined goal. In a game field where obstacles and undulations arearranged, the game control block 311 moves the player's character 516 inaccordance with the input from the direction key or analog sticks or thelike of the input apparatus 6 and makes the player's character 516 jumpin accordance with the input from a button or the like.

In such an action game, depending on the position of a game field,different methods are applied to the setting of the viewpoint positionof the camera used by the first image generation block 316 whengenerating the first game image. In the example of this diagram, theviewpoint position is set at a predetermined position in the rear andabove the player's character 516 and the viewpoint position is moved asthe player's character 516 moves. The sightline direction is changed inaccordance with the attitude of the head mounted display 100. Namely, animage is generated in which the user feels as if he or she were viewingthe game field riding a vehicle following the player's character 516from behind.

Referring to FIG 14, there is shown an example of the first game imagethat is displayed on the head mounted display 100. In the example ofthis diagram, the viewpoint position is fixed to a predeterminedposition in a game field and only the sightline direction is changed inaccordance with the attitude of the head mounted display 100. Namely, animage is generated in which the user feels as if he or she were viewingthe game field without moving after getting down from a vehicle.

For the data of a game field, data for specifying rules of determining aviewpoint position is set in addition to contour data such as geologicalfeatures and the data of arranged characters and items. These pieces ofdata may be set for each area obtained by dividing the game field intotwo or more areas. Updating the position of the player's character inaccordance with the instruction input by the user, the game controlblock 311 transmits the rules of determining the viewpoint positionspecified at the updated position to the first image generation block316. The first image generation block 316 determines the viewpointposition in accordance with the rules transmitted from the game controlblock 311, thereby rendering the game field. For example, in the exampleshown in FIG. 14, the viewpoint position remains fixed while theplayer's character is inside the predetermined area; however, when theposition of the player's character deviates from the predetermined areaand enters another area, the first image generation block 316 determinesthe viewpoint position in accordance with the rules specified in the newarea, thereby setting the viewpoint position at a predetermined positionin the rear of the player's character for example.

Consequently, a game producer can flexibly set how to suitably generategame images by appropriately setting a viewpoint position in accordancewith the geological features of a game field, the types of arrangeditems, gimmicks, and characters, and the actions to be executed by aplayer's character. Further, since appropriate game images can begenerated in match with the situations of, a game field, userconvenience is enhanced.

The rules of determining a viewpoint position may be dynamically set inaccordance with situations of a game field. For example, in the casewhere the geological features of the game field change, an algorithm fordetermining rules from the geological features of the game field may bespecified in advance, thereby first determining the rules from thegeological features of the game field and then determining a viewpointposition in accordance with the determined rules. Consequently, anappropriate viewpoint position can be set in conformity with thesituations of the game field, thereby enhancing user convenience.

Match Game:

Referring to FIG. 15, there is shown an example of the first game imagethat is displayed on the head mounted display 100. This diagram shows animage of a game in which a first user wearing the head mounted display100 and a second user viewing the display apparatus 12 fight each other.In this match game, while throwing a bomb 522 to the second character520 of the second user by the input in the input apparatus 6, the firstuser moves the head mounted display 100 to move a first character,thereby avoiding a bomb 524 thrown by the second character 520. Whilethrowing the bomb 524 to the first character by the input in the inputapparatus 6, the second user enters the input in the input apparatus 6to move the second character 520, thereby avoiding the bomb 522 thrownby the first character. Two or more second users may participate in amatch game; in this case, two or more second characters may attack thefirst character.

The first image generation block 316 sets a viewpoint position at aposition of the first character in a game field and then sets thesightline direction in the direction to which the head mounted display100 faces, thereby generating the first image. Therefore, in the exampleshown in this diagram, a game image of first-person viewpoint isdisplayed on the head mounted display 100. The bomb 524 thrown by thesecond character 520 looks like it is flying to the user, so that a gamehigh in reality can be provided.

The game control block 311 changes the position of the first characteras the attitude of the head mounted display 100 changes. In addition,the game control block 311 throws the bomb 522 in the directionindicated by the first character in accordance with the instructioninput to the input apparatus 6 and computes a locus by physicalcomputation or the like, thereby moving the bomb 522. If the bomb 522thrown by the first character hits the second character 520, then thephysical strength value of the second character 520 is decreased by apredetermined value. The bomb 524 thrown by the second character 520hits the first character, then the physical strength value of the firstcharacter is decreased by a predetermined value. If the physicalstrength of any one of the characters reaches zero, the game ends atthat moment.

Referring to FIG. 16, there is shown an example of the second game imagethat is displayed on the display apparatus 12. The second imagegeneration block 317 fixes the viewpoint position and the sightlinedirection so as to allow the view over the entire game field and thedisplaying of a first character 526 and the second character 520 in thegame image, thereby generating the second image. If there are a singlesecond user and a single second character 520, then the second imagegeneration block 317 may set the viewpoint position at the position ofthe second character 520 or a predetermined position in the rear of thesecond character 520 to set the sightline direction in the movingdirection of the second character 520, thereby generating the secondimage; if there are two or more second users and two or more secondcharacters 520, then it is desirable for the second image generationblock 317 to generate the second image fixed such that the entire gamefield can be viewed, thereby facilitating all second users to view thegame image.

In accordance with the instruction input to the input apparatus 6, thegame control block 311 changes the position of the second character 520.In addition, in accordance with the instruction input to the inputapparatus 6, the game control block 311 throws the bomb 524 in thedirection indicated by the second character 520 and computes a locus byphysical computation or the like, thereby moving the bomb 524.

As described above, in a game where the first user and the second usershare the same virtual three-dimensional space and both the usersinteract with each other in this virtual three-dimensional space, gameimages with the same virtual three-dimensional space viewed fromseparate viewpoints can be provided to each of the users.

Referring to FIG. 17, there is shown a flowchart indicative of aprocedure of a display control method practiced as one embodiment of thepresent invention. The HMD information acquisition block 314 acquiresthe positional information of the head mounted display 100 obtained froma taken image of the head mounted display 100 (S100) and the inputapparatus information acquisition block 315 acquires the positionalinformation of the input apparatus 6 acquired from a taken image of theinput apparatus 6 (S102). The instruction input acquisition block 312acquires the information associated with the instruction input from theinput apparatus 6 (S104). The game control block 311 controls the gameon the basis of the position of the head mounted display 100, theposition of the input apparatus 6, and instruction input from the inputapparatus 6 (S106). The first image generation block 316 generates animage of the game to be controlled by the game control block 311, theimage being to be displayed on the head mounted display 100 (S108). Atthis moment, the first image generation block 316 displays the image ofthe input apparatus 6 in the generated image on the basis of a relativeposition between the. head mounted display 100 and the input apparatus 6(S110). The generated image is transmitted to the head mounted display100 (S112).

Referring to FIG. 18, there is shown a flowchart indicative of aprocedure of the display control method practiced as one embodiment ofthe present invention. The instruction input acquisition block 312acquires the information associated with an instruction input from theinput apparatus 6 (S120). The game control block 311 updates theposition subject to operation on the basis of an input instruction(S122) and acquires rules for determining a viewpoint position specifiedat the updated position (S124). The first image generation block 316determines the viewpoint position in accordance with the rules fordetermining a viewpoint position (S126). The HMD information acquisitionblock 314 acquires the information associated with the attitude of thehead mounted display 100 from the head mounted display 100 (S128). Thefirst image generation block 316 generates an image to be displayed onthe head mounted display 100 by use of the determined viewpoint positionand the sightline direction determined on the basis of the attitude ofthe head mounted display 100 (S130). The generated image is transmittedto the head mounted display 100 (S132).

Referring to FIG. 19, there is shown a flowchart indicative of aprocedure of the display control method practiced as one embodiment ofthe present invention. The instruction input acquisition block 312acquires the information associated with an instruction input from theinput apparatus 6 of the first user (S140) and, at the same time,acquires the information associated with an instruction input from theinput apparatus 6 of the second user (S142). The HMD informationacquisition block 314 acquires the information associated with positionand attitude of the head mounted display 100 (S144) and the inputapparatus information acquisition block 315 acquires the informationassociated with the position and attitude of the input apparatus 6(S146). The game control block 311 controls the game on the basis of theinput instruction, the position and attitude of the head mounted display100, and the position and attitude of the input apparatus 6 (S148). Thefirst image generation block 316 generates the first image to bedisplayed on the head mounted display 100 by use of the viewpointposition and sightline direction of the first user (S150), the generatedfirst image being transmitted to the head mounted display 100 (S152).The second image generation block 317 generates the second image to bedisplayed on the display apparatus 12 by use of the viewpoint positionand sightline direction of the second user (S154), the generated secondimage being transmitted to the display apparatus 12 (S156).

While preferred embodiments of the present invention have been describedusing specific terms, such description is for illustrative purpose only,and it is to be understood by those skilled in the art that changes andvariations may be made without departing from the spirit or scope of thefollowing claims.

In the examples described above, an image for monocular vision isdisplayed on the display apparatus 190 of the head mounted display 100;it is also practicable to display an image for binocular vision.

In the examples described above, the head mounted display 100 is used ina game system; the technologies described in the embodiments of thepresent invention are also applicable to display content other thangames.

The present technology contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2015-039075 filed in theJapan Patent Office on Feb. 27, 2015, the entire content of which ishereby incorporated by reference.

What is claimed is:
 1. A display control apparatus comprising: a headmounted display information acquisition block configured to acquireinformation indicative of a position of a head mounted display worn on ahead of a user; an input apparatus information acquisition blockconfigured to acquire information indicative of a position of an Inputapparatus for use by the user; an instruction input acquisition blockconfigured to acquire information indicative of an instruction enteredin the input apparatus; and an image generation block configured togenerate a image to be displayed on the head mounted display on a basisof an instruction acquired by the instruction input acquisition block,wherein the image generation block displays an image of the inputapparatus at a position in the image determined on a basis of a relativeposition with the head mounted display of the input apparatus.
 2. Thedisplay control apparatus according to claim 1, wherein the imagegeneration block changes a position of the input apparatus that isdisplayed in the image as a position of, the input apparatus changesand, if the position of the input apparatus in the Image deviates from apredetermined range, notifies the user thereof.
 3. The display controlapparatus according to claim 1, wherein the input apparatus informationacquisition block acquires information associated with an attitude ofthe input apparatus and, on a basis of the attitude of the inputapparatus, the image generation block changes the attitude of the inputapparatus to be displayed in the image.
 4. The display control apparatusaccording to claim 1, wherein, on a basis of information indicative ofan instruction entered in the input apparatus acquired by theinstruction input acquisition block, the image generation block changesan image of the input apparatus displayed in the image.
 5. The displaycontrol apparatus according to claim 1, wherein the head mounted displayinformation acquisition block further acquires information associatedwith an attitude of the head mounted display, and the image generationblock determines a sightline direction for generating the image on abasis of the attitude of the head mounted display.
 6. The displaycontrol apparatus according to claim 1, wherein the input apparatusinformation acquisition block acquires information associated with aposition of the input apparatus acquired on a basis of an image of theinput apparatus taken by an imaging apparatus and, if the inputapparatus is not taken by the imaging apparatus and thereforeinformation associated with a position of the input apparatus is notacquired, then the input apparatus information acquisition blockacquires a position of the input apparatus on the basis of informationassociated with an attitude of the input apparatus.
 7. A display controlmethod for a computer, comprising: acquiring information indicative of aposition of a head mounted display worn on a head of a user; acquiringinformation indicative of a position of an input apparatus for use bythe user; acquiring information indicative of an instruction entered inthe input apparatus; generating an image to be displayed on the headmounted display on the basis of an acquired instruction; and displayingan image of the input apparatus at a position in the image determined ona basis of a relative position with the head mounted display of theinput apparatus.
 8. A non-transitory computer-readable recording mediumrecording a display control program for a computer, the programcomprising: by a head mounted display information acquisition block,acquiring information indicative of a position of a head mounted displayworn on a head of a user; by an input apparatus information acquisitionblock, acquiring information indicative of a position of an inputapparatus for use by the user; by an instruction input acquisitionblock, acquiring information indicative of an instruction entered in theinput apparatus; and by an image generation block, generating an imageto be displayed on the head mounted display on a basis of an instructionacquired by the instruction input acquisition block, wherein the imagegeneration block displays an image of the input apparatus at a positionin the image determined on a basis of a relative position with the headmounted display of the input apparatus.